A mathematical technique to minimize loss . Gradient descent iteratively adjusts weights and biases , gradually finding the best combination to minimize loss. Gradient descent is older—much, much older—than machine learning .1
목차
Algorithm
The model begins training with randomized weights and biases near zero, and then repeats the following steps:2
Calculate the loss with the current weight and bias.
Determine the direction to move the weights and bias that reduce loss.
Move the weight and bias values a small amount in the direction that reduces loss.
Return to step one and repeat the process until it converges
Footnotes
목차 관련 글 ML crash course - Linear regression 이 글을 인용: “Define and describe how gradient descent finds the optimal model parameters.” Nested Learning: The Illusion of Deep Learning Architectures 이 글을 인용: “In addition to its neuroscientifically plausible and mathematically white-box nature, we advocate for its impor…” 역전파 이 글을 인용: “The algorithm that implements gradient descent in neural networks.” Learning rate 이 글을 인용: “A floating-point number that tells the gradient descent algorithm how strongly to adjust weights and biases on …” Convex function 이 글을 인용: “Gradient descent: Many variations of gradient descent are guaranteed to find a point close to the minimum of a …” Clipping (machine learning) “Gradient clipping forces gradient values within a designated range during training.” Vanishing gradient problem “Models suffering from the vanishing gradient problem become difficult or impossible to train.” mini-batch “Therefore, each iteration determines the loss on a random 20 of the 1,000 examples and then adjusts the weights…” 배치 크기 “Mini-batch stochastic gradient descent, in which the batch size is usually between 10 and 1000.” Convex optimization 이 글을 인용: “The process of using mathematical techniques such as gradient descent to find the minimum of a convex function.” Exploding gradient problem “The tendency for gradient in DNNs (especially RNNs) to become surprisingly steep (high).” Introduction to Machine Learning 관련 절: Definition of [[Machine learning|ML]] Gradient boosting “Gradient boosting is a powerful ensemble machine learning technique used for both regression and classification…” Gradient (machine learning) “In machine learning, the gradient is the vector of partial derivatives of the model function.” Stochastic gradient descent 이 글을 인용: “A gradient descent algorithm in which the batch size is one.” Iteration (machine learning) “A backward pass (backpropagation) to adjust the model's parameters based on the loss and the learning rate.” AdaGrad 이 글을 인용: “A sophisticated gradient descent algorithm that rescales the gradients of each parameter, effectively giving ea…” Gradient clipping 이 글을 인용: “A commonly used mechanism to mitigate the exploding gradient problem by artificially limiting (clipping) the ma…” Training loss “A loss curve provides the following hints about training:” Optimizer (machine learning) 이 글을 인용: “A specific implementation of the gradient descent algorithm.”